fix(relinker): agree on the syscall number between the two constant-load paths - #10439

Reviewed head e8fd90f merged onto upstream/main aeeaefa (2026-10-24); the merge is clean, and so are merges onto the current main 50aa179 and together with #10479 and #10537.

Environment: Windows 11 Home 10.0.26200, WinLibs GCC 15.2.0 UCRT, CMake 4.3.1, Ninja 1.13.2, Python 3.13.9, -DANYPS5_RELINKER_ONLY=ON -DBUILD_TESTING=ON, Release. My machine was under heavy unrelated CPU load, so I report targeted tests only; the relinker matrix is green on the PR.

Results

• Merged tree: syscall_number, sce_code_gaps, guest_syscall_trap, sdk_mman_syscall pass. Red/green: with SyscallScanner.cpp reverted and the PR's tests kept, syscall_number fails with ('sign-extended', 'did not report syscall number 18446744073709551615'); restored, it passes. With #10479 and #10537 merged on top, syscall_number and sce_code_gaps (including #10537's new cases) still pass. The C++ syscall_scanner is Linux-only in CI and was not run here; the Python test carries the same byte sequences.
• Encodings checked with objdump -M intel: 48 c7 c0 ff ff ff ff = mov rax, 0xffffffffffffffff (sign-extended), 48 31 c0 = xor rax, rax, 66 31 c0 = xor ax, ax, 45 31 c0 = xor r8d, r8d, 44 31 c0 = xor eax, r8d. Every form the PR accepts does set rax to the reported constant; I found no over-acceptance.
• check_conventions.py passes; pr_overlap.py lists only #8042 as not compared (one unrelated line, and #8042 no longer merges against main).

Should fix (text)

• "Every form the decoded walk newly recognizes was already recognized by the byte window, so which conversions are refused or skipped is unchanged" does not hold. Outside the unwind functions a gap pattern is skipped only when the decoded walk returns no number (SyscallScanner.cpp:302-313), and the decoded walk steps over F2/F3 prefixes that the byte window (MovesOtherRegisterThanRax, ConstantRaxSetEndingAt) cannot. With test_sce_code_gaps.executable and --skip-sce-module --registry: 48 B8 09 00 00 00 00 00 00 00 F2 49 89 CA 0F 05 C3 exits 0 on main ("Skipped 1 syscall byte pattern(s) ... 0x21e") and exits 2 on this branch ("Forbidden syscall instruction at code offset 0x21e: syscall number 9"); 31 C0 F3 0F 05 C3 likewise flips from skipped (0x212) to refused with number 0; the unprefixed 48 B8 imm64; 49 89 CA; 0F 05 and the 48 C7 C0 forms are refused on both. The change is stricter, not looser, which is fine, but the body should say that gap patterns with an F2/F3-prefixed mov r10, rcx or a prefixed syscall after a 48 B8 imm64 or 31/33 C0 load are now refused where main skipped them.

Nits

• Body: "the sign-extended value no longer truncates through std::optional's converting constructor" misdescribes main: ConstantRaxLoad returned a std::uint32_t read without sign-extension (SyscallScanner.cpp:131-139 on main), and the only optional conversion (number = ConstantSyscallNumber(...) into optional<uint64_t>, line 304) widened. The widened return type is what lets the sign-extended value through; nothing truncated before.
• kExtendedSourceZeroedStub / "stub that zeroes eax from r8d" (SyscallScannerTests.cpp:124 and :207, and extended-source-zeroed in the Python test): 44 31 C0 is xor eax, r8d (31 /r is XOR r/m32, r32; REX.R extends the source), eax becomes eax ^ r8d; the expected "no number" is right, the name is not.
• The 33 C0 arm is accepted but never tested: 48 33 C0 (0) and a negative control such as 44 33 C0 = xor r8d, eax (no number) would pin it.

Notes

• The decoded path still rejects some legal constant loads: 40 31 C0 (REX with no bits), 66 48 31 C0 (REX.W overrides 66), 4C C7 C0 imm32 and 44 B8 imm32 (REX.R is ignored for these); consequence is only a missing number in the message, and compilers do not emit them.
• Cross-PR: #10537 replaces exactly the byte-window 66 31 C0 arm this body says is left alone, and it reports number 0 for 44 31 C0 and 41 33 C0, where this PR's decoded path (correctly) reports none; after both merge the two paths disagree on those bytes. I have left a note on #10537. Whichever of the two lands second should restate the resulting byte-window behaviour in its body.

Not checked: the macOS counts (231/231); the C++ syscall_scanner; a full suite; two consecutive REX bytes (48 48 31 C0 is reported as 0 by the PR; I did not verify the hardware semantics).

I found no blocking issue: the encodings are right, the counter-check holds, and the fix does what #10422 asks. The one thing to settle before merge is the body's claim about unchanged refuse/skip outcomes. For disclosure: I have open relinker PRs (#10564, #10402, #10401, #10403, #10390, #10391); none touches SyscallScanner.cpp.

Written with AI assistance (Claude Code). I ran the builds, tests and fixture checks above on my Windows 11 machine.

Follow-up on exact head e8fd90f4e0533d6b3a760336f9957644b700842a, considering the current #10537 head 1f6c936b36446eeadc0c1e0f8bd37a4de8c65aa0. #10537 expands the byte-window recognizer, so the two paths now disagree on additional valid register encodings relevant to #10422:

• 42 31 C0 is xor eax,eax (REX.X does not extend either register field). The byte window now reports 0, but ConstantRaxLoad rejects every REX value except 0x48, so the decoded path reports no syscall number.
• 47 31 C0 is xor r8d,r8d. The byte window skips this write and can continue to a preceding mov eax,imm32; the decoded backward walk only skips 49 89 CA, so it stops at this unrelated write and reports no number instead of the loaded value.

This reintroduces the same path-dependent diagnostic the issue is meant to remove. Please align the decoded path with the prefix-aware byte recognizer (and add differential cases covering both unwind-backed and gap/fallback paths), or coordinate an alternative that makes both paths report the same value. My review of #10537's current build ran ctest --test-dir build -R '^(sce_code_gaps|syscall_number)$' --output-on-failure: 2/2 passed, but those tests do not compare the same encodings across both paths.

Re-reviewed exact head ad6fa74e4ac99bafe2e33605360d51ee75ba12bc. The head delta from e8fd90f4 only renames an XOR test; the path disagreement relevant to #10422 remains. I built this exact head with GCC 12.2 in a relinker-only Release configuration. ctest -R '^(sce_code_gaps|syscall_number|syscall_scanner)$' --output-on-failure passed 3/3, but two extra exact-head fixtures reproduced the uncovered bug: decoded 42 31 C0 has no number while the SCE-gap scan reports 0; a gap fixture with mov eax,59; 47 31 C0 reports 0 though the XOR only writes R8D and should preserve 59. git diff --check upstream/main...HEAD and the exact-head conventions check pass. Hosted relinker matrix and ASan pass; product jobs are skipped. Please resolve the decoded/byte-path mismatch and add differential coverage before merge.

Reviewed head 0832fe91b merged onto main 0119103fa (clean; still clean on 79f061ba5, which touches none of these files), and that merge plus #10537 1f6c936b. Linux: WSL2 Ubuntu 24.04, GCC 13.3, relinker-only Release.

What I ran

• ctest -R '^(sce_code_gaps|syscall_number|syscall_scanner)$': 3/3 pass on the merge and on the merge with #10537. With main's SyscallScanner.cpp and your tests kept, all 3 fail (mov rax, -1 gives 4294967295, prefixed-zeroed is skipped). So the C++ syscall_scanner now has a Linux red/green too.
• MarcZX10's two cases are fixed: decoded 42 31 c0 gives 0, and the gap mov eax,59; 47 31 c0 gives 59.
• roboterspinne's rows, each through the decoded fixture, a gap, and the unwind function. With #10537 the decoded and gap answers match on every row except 66 48 31 c0 and mov eax,59; 41 b0 05. Those are the gap-only cases the body names.

Should fix: the new step-over accepts instructions that write eax

ConstantSyscallNumber now calls MovesOtherRegisterThanRax(bytes, match.Length) on decoded instructions of any length (SyscallScanner.cpp:194). That helper was written for 2- or 3-byte raw windows. For any other length it sets rex to 0 and takes the last two bytes as opcode and ModRM (:161-167). So any 4+ byte instruction whose tail looks like 89/8B/31/33 + a mod-11 byte gets stepped over, even when it writes eax.

Example: b8 3b 00 00 00 8b 44 8b f8 0f 05 c3. objdump gives mov eax,0x3b; mov eax,DWORD PTR [rbx+rcx*4-0x8]; syscall. The tail 8b f8 reads as mov edi, eax.

• main: no number in the decoded fixture and inside the function; the gap is skipped.
• this PR: syscall number 59 on all three paths.

Same with 8b 44 33 f8 (mov eax,[rbx+rsi-8]), 05 00 00 31 c9 (add eax,0xc9310000) and 66 b8 89 c1 (mov ax,0xc189). The last one also creates a new disagreement in a gap. There main's byte window gives the right 49545, but the decoded walk now runs first and names 59. #10537 doesn't change any of these.

Here's what I tried locally: step over only when the instruction is just (REX) + opcode + ModRM, and pass the helper the slice that starts at the REX byte:

const std::size_t opcode = Codegen::DecodedInstruction{bytes, match.Length}.OpcodeOffset();
const std::size_t begin = opcode > 0 && (bytes[opcode - 1] & 0xF0) == 0x40 ? opcode - 1 : opcode;
if (match.Length - opcode != 2 || !MovesOtherRegisterThanRax(bytes + begin, match.Length - begin))
    return std::nullopt;

With that, the 3 tests still pass and the four sequences above get main's answers back (no number, skipped, 49545 in the gap). Every roboterspinne row gives the same answer as on your head. f2 49 89 ca still steps over, but now because REX.B makes it r10. Today it passes only because the dropped REX leaves rm 2 (rdx). A no-number case like b8 3b 00 00 00 8b 44 8b f8 0f 05 c3 would pin this.

Nits

• kExtendedLoadStub is 41 B8, which is mov r8d. Its description says "loads r9d".
• Body: "steps over every register-to-register mov/xor whose REX, opcode and ModRM bytes MovesOtherRegisterThanRax accepts" holds only for 2- and 3-byte instructions right now (see above).

Checked and fine: every ConstantRaxLoad form against the SDM encodings. 48 C7 C0 sign-extends. B8 with W=0 and B=0 zero-extends. 31/33 C0 with R=0 and B=0, without 66, zeroes rax. A REX before a legacy prefix is ignored. I found no over-acceptance there.

Not checked: Windows and macOS, the full suite, real binaries.

@KSWORK2001 Thanks, checked 0f45390 merged onto main 204c674 (2026-10-26); the merge is clean. Windows 11, GCC 15.2, relinker-only Release.

• Full ctest: 306 of 307 pass; patch_advisor_unit fails the same way on plain main, and long_paths is skipped. syscall_number, sce_code_gaps, sdk_mman_syscall, guest_syscall_trap and input_magic pass.
• Counter-check against the new tests:
  • The previous head's SyscallScanner.cpp fails syscall_number at load-then-memory-load and sce_code_gaps at low-word-ending-in-move; both report 59.
  • main's fails at sign-extended and prefixed-zeroed.
• The decoys from the last round get main's answers back: no number, or a skipped gap, and 49545 for 66 b8 89 c1 in a gap. After mov eax, 59, 89 c1, f2 49 89 ca, f2 49 89 c8 and 66 89 c1 report 59, and the mprotect tail reports 74.
• The hosted Build passed on Linux, macOS and Windows.
• It still merges cleanly with #10537. With #10479 there is still one conflict in test_syscall_number.py, where both add constants after PREFIXED_CLOBBERED; whichever lands second keeps both blocks.

That settles the points from my review and my follow-up; I found nothing else to change.

Written with AI assistance (Claude Code). I ran the checks above on my Windows 11 machine.